Video production systems and methods

ABSTRACT

Live video streams are produced using a network server system. The network server system initially receives, via a network, one or more captured video streams that are live streams captured from a remotely-located camera, phone or the like. The captured streams are forwarded to a control device via the network to thereby permit a user of the control device to select one of the captured video streams for output to the video production stream. In response to a command received from the control device that indicates the selected capture stream, the selected capture video stream is encoded for output as the video production stream.

PRIORITY CLAIM

This application is a continuation of U.S. patent application Ser. No.16/024,775 filed on Jun. 30, 2018, now U.S. Pat. No. 11,363,311. Thatapplication claims priority to India Provisional Application Serial No.201841001038 filed on Jan. 10, 2018.

TECHNICAL FIELD

The following discussion generally relates to data communication,particularly wireless data communication. More particularly, thefollowing subject matter relates to network-based video productionsystems.

BACKGROUND

Recent years have seen a dramatic increase in the use of mobile devicesin conjunction with wireless networks, such as WiFi networks. At thesame time, there has been a marked increase in the creation andenjoyment of digital video content. This is in part due to the fact thatmillions of people around the world now carry mobile phones, cameras orother devices that are capable of capturing high quality video and/or ofplaying back video streams in a convenient manner.

Conventional video production systems typically used a dedicatedhardware device to perform on-site video encoding. One example of avideo production system that uses a local encoder device is described inU.S. patent application Ser. No. 15/920,338, which is incorporated byreference herein as one example of a video encoding system. In thatexample, various “camera operators” capture live video content with amobile phone or other camera that is streamed to a dedicated encodingdevice located on-site where the video is being captured. Typically, theencoder is located within range of a Wi-fi or similar wireless networkto permit live streaming of captured video directly from the camera tothe encoder. On-site encoding has the benefit of high bandwidth localstreaming and dedicated encoding hardware. Several drawbacks can exist,however. The need to purchase and maintain dedicated encoding hardwaremay be prohibitive (or at least inconvenient) for some users,particularly those users who may have limited use for dedicated encodinghardware.

In some contexts in which a processing device—e.g., a video processingdevice adapted to receive video content from multiple mobile devices or“clients” over a wireless network via an access point—it is often thecase that a number of other access points might be operating in the samearea and receiving/transmitting packets within the same channel(s) asthe processing device. The resulting competition over resources (i.e.,between access points and the mobile devices themselves) can negativelyaffect the bandwidth available for data transmission between theindividual mobile devices and the processing device, particularly in thecase of the transmission of video and other high-bandwidth data.

It is therefore desirable to create systems and methods for dynamicallyproducing video content from multiple client devices. Other desirablefeatures and characteristics will become apparent from the subsequentdetailed description and the appended claims, taken in conjunction withthe accompanying drawings and this background section.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

Example embodiments will hereinafter be described in conjunction withthe following drawing figures, wherein like numerals denote likeelements, and:

FIG. 1 is a diagram of an example system for encoding, producing anddistributing live video content in accordance with one embodiment.

FIG. 2 presents a flowchart illustrating a method in accordance with oneembodiment.

BRIEF DESCRIPTION

Various embodiments replace or supplement on-site video encoding duringlive video production with network-based or similarly remote processingperformed by a network server, cloud service and/or the like.

In an example embodiment, an automated process executed by a dataprocessing system to create video production stream. The automatedprocess suitably comprises: receiving, via a network, a plurality ofcaptured video streams, wherein each of the captured video stream is alive stream received via a network from one of a plurality ofremotely-operated capture devices; forwarding each of the captured videostreams to a control device via the network to thereby permit a user ofthe control device to select one of the captured video streams foroutput to the video production stream; and, in response to a commandreceived from the control device, encoding the selected one of thecaptured video streams for output to the video production stream.

Other embodiments relate to network server systems and devices, controldevices, video capture devices and/or automated processes performed byany of such systems or devices. These and other embodiments aredescribed below.

DETAILED DESCRIPTION

The following detailed description of the invention is intended toprovide various examples, but it is not intended to limit the inventionor the application and uses of the invention. Furthermore, there is nointention to be bound by any theory presented in the precedingbackground or the following detailed description.

The systems and methods described herein may be used in a variety ofcontexts. Without loss of generality, the various systems and methodsare often described in the context of a video production system in whichone or more live video streams are received via a wireless network toproduce an output video stream for publication or other sharing. Theembodiments are not so limited, however. Any network in which multipleclient devices are connected to a common access point may utilize thedisclosed methods and systems.

According to various embodiments, on-site encoding hardware is replacedand/or supplemented with cloud-type encoding resources that areavailable via the Internet or another wide area network (WAN). Thereduction (or elimination) of on-site hardware allows for morewidespread adoption by different types of customers, including morecasual users who may not have the desire or resources to purchasededicated hardware. To that end, network processing resources availablefrom a dedicated server and/or shared cloud processing (e.g., Amazon WebServices (AWS), Microsoft Azure services and/or the like).

FIG. 1 illustrates an example of a video production system (or simply“system”) 100 that might be used to produce a video program based uponselected inputs from multiple input video feeds. In the illustratedexample, system 100 includes a video processing module 152 (incorporatedvia appropriate software and/or hardware within a server 150 or thelike) that selects and encodes video content based on instructionsreceived from a control device 130 via network 105. The encoded videoprogram may be initially stored as a file within an external storagedevice (e.g., a memory card, hard drive or other non-volatile storage)for eventual uploading to a hosting, distribution, and/or processingserver 150 operating on the Internet or another network 105. In someembodiments, the encoded video program is directly streamed or otherwisetransmitted to a social media platform or video hosting service forviewing by the public, friends, or some other selected subset ofindividuals.

Processing module 152 includes suitable processing hardware such as amicroprocessor, a memory, and input/output interfaces (including, forexample, a suitable USB or other interface to the external storage).Processing module 152 includes logic (implemented as hardware and/orsoftware) for communicating with any number of clients 160, illustratedas an array of video capture devices 160A-F connected to network 105 viaone or more access points (such as IEEE 802.11 APs 161 and 162).

Clients 160 are any devices capable of capturing live video content andproviding a stream of the captured content to server 150 via theInternet or another wide area network (WAN) 105. Clients 160 may includemobile phones, tablets, laptops or similar devices executing a videocapture application 162, as desired. Clients 160 might also include oneor more conventional video cameras 164 that interact with processingmodule 152 via an interface device that receives DVI or other videoinputs and transmits the received video to the processing module 152 viaa Wi-fi, Bluetooth or other wireless network, as appropriate. Otherembodiments could facilitate communications with any other types ofvideo capture devices in any other manner.

Processing module 152 may be implemented as a control applicationexecuting on a processor within server iso, for example, that includeslogic for implementing the various processes described herein. As notedabove, server 150 may be implemented with conventional computinghardware, including any number of processors, data storage, datainterfaces and the like. In some embodiments, server 150 is implementedwithin the context of a web service system that includes conventionalload balancing, firewalls, user authentication, network administrationand control and the like. Equivalently, server 150 may be implementedusing any so-called “cloud” service that abstracts the actual hardwareperforming the various functions, such as the AWS service available fromAmazon.com, the Azure service available from Microsoft Corp., or anyother service as desired.

Other embodiments may implement the various functions and features usinghardware, software and/or firmware logic executing on other components,as desired. An encoder 116, for example, may be implemented using adedicated video encoder chip in some embodiments.

In various embodiments, processing module 152 operates in response touser inputs supplied by an operator of a control device 130. Controldevice 130 may correspond to any type of computing device that includesa processor 131, memory 132 and input/output feature 133. In variousembodiments, control device 130 is as a tablet, laptop or other computersystem, for example, or a mobile phone or other computing device thatexecutes a software application 140 for controlling the functions ofsystem 100.

The example illustrated in FIG. 1 depicts control application 140 havingan interface that shows various video feeds received from imagecollection devices 160A-F and lets the user select an appropriate feedto encode into the finished product. Application 140 may include otherdisplays to control other behaviors or features of system 100, asdesired. Typically, control device 130 interacts with processing module152 via network 105. Control device 130 may access the network 105 via amobile telephone or other wireless network, although wired connectionscould be equivalently used.

In operation, referring now to FIG. 2, video feeds are received bymodule 152 at server 150 from one or more capture devices 160 (step201). Some or all of the video feeds received from the capture devices160 are then provided by server 150 to control device 130 via network105. Video streams supplied from server 150 to control device 130 may besubstantially compressed to preserve bandwidth and reduce latency. Invarious embodiments, the various capture feeds are combined into asingle output stream encoded by server 150 so that only one stream needsto be provided to the control device 130. In further embodiments, thestream is compressed to the point of still imagery (e.g., i-frames) thatare sent on any regular or irregular temporal basis so that the controldevice 130 simply receives frequent “snapshots” of the various capturefeeds rather than full motion video. The imagery may be furthercompressed or combined with other captured imagery or video as desired.

Typically, a user acting as a video producer uses application 140executing on control device 130 to view the various capture feeds thatare available, and to select one of the feeds for an output stream asdesired. Application 140 notifies server 150 of the selected feed, andserver 150 suitably processes and/or otherwise manipulates the variousvideo feeds that are available from one or more capture devices 160A-Fas appropriate (step 202).

As noted above, the selected video feed is selected as one of the feedsreceived from the clients 160. The video processing module 152 suitablycompresses or otherwise encodes the selected video in an appropriateformat for eventual viewing or distribution (steps 203-204). Videoencoding by module 152 occurs according to any standard, non-standard orother protocol, such as MPEG4. Updated captured video feeds are providedto the control device as appropriate so that the producer can selectdifferent feeds as needed to create a desired output stream. Differentcapture devices located at different vantage points, for example, may beselected at different times to provide desired views of the event beingcaptured.

At any location, it is possible that there are multiple APs operating inthe same or different Wi-Fi channels (such as APs 161 and 162). APs thatoperate on different channel numbers will not interfere with each other,while APs that operate on the same channel will interfere with eachother and result in reduction of bandwidth/throughput. If two or moreAPs are operating via the same channel number, then all the APs sharethe available WiFi medium in a “half-duplex” manner. For example, if AP115 and AP 161 operate via channel 155, then at any point either of theAPs can send data, but not simultaneously.

Various embodiments therefore allow remote processing capabilities tosupplement or replace on-site encoding during live video production.Feeds captured from phones, cameras or the like can be sent to a networkserver for initial processing (e.g., compression, sampling ordecimation) before being sent to a control device for review by a videoproducer. The producer selects one of the captured feeds to be used inan output stream, and the control device notifies the network server ofthe selected stream. The network server responsively encodes theselected stream in an appropriate output format for distribution,storage or the like. Various embodiments are therefore able to performreal-time video production using live-captured imagery from one or morecameras. “Real-time” in this context is intended to recognize that somedelay is inherent in data transmission and processing.

The term “exemplary” is used herein to represent one example, instanceor illustration that may have any number of alternates. Anyimplementation described herein as “exemplary” should not necessarily beconstrued as preferred or advantageous over other implementations. Whileseveral exemplary embodiments have been presented in the foregoingdetailed description, it should be appreciated that a vast number ofalternate but equivalent variations exist, and the examples presentedherein are not intended to limit the scope, applicability, orconfiguration of the invention in any way. To the contrary, variouschanges may be made in the function and arrangement of the variousfeatures described herein without departing from the scope of the claimsand their legal equivalents.

What is claimed is:
 1. An automated process executed by a dataprocessing system accessible via a network to create a video productionstream, the automated process comprising: receiving, via the network, aplurality of captured video streams, wherein each captured video streamis a live stream of an event, and wherein each captured video stream isreceived by the data processing system via a network from one of aplurality of remotely-operated capture devices; combining, by the dataprocessing system, the video imagery from each of the captured videostreams received from the plurality of capture devices to create asingle combined stream that includes video imagery from each of thecaptured video streams; forwarding the single combined stream thatincludes the imagery from each of the captured video streams from thedata processing system via the network as an input stream provided to acontrol device located remotely from the data processing system tothereby permit a user of the control device to view each of the capturedvideo streams within the input stream and to select one of the capturedvideo streams present in the single combined stream; and in response toa command received via the network from the control device thatidentifies the selected one of the captured video streams from thesingle combined stream, the data processing system selecting theselected one of the captured video streams from the captured videostreams received from the plurality of remotely-operated capture devicesand providing only the selected one of the captured video streams foroutput to the video production stream.
 2. The automated process of claim1 wherein the data processing system is a cloud-based video processingsystem operating on hardware that is not located at a location of theevent, and wherein each of the captured video streams are captured bythe plurality of remotely-operated capture devices each located on-siteat the location of the event.
 3. The automated process of claim 2wherein the captured video streams are received by the data processingsystem in real time.
 4. The automated process of claim 3 wherein theencoding is performed in real time to create the video production streamas a live stream from the captured video streams received from theplurality of remotely-operated capture devices at the location of theevent.
 5. The automated process of claim 4 wherein control device islocated on-site at the location of the event, and wherein the singlecombined stream is forwarded to the control device in real time
 6. Theautomated process of claim 5 further comprising the data processingsystem compressing the captured video streams in real time prior toforwarding the single combined stream to the control device located atthe location of the event.
 7. The automated process of claim 5 whereinimagery provided in the single combined stream forwarded to the controldevice comprises a series of still images from each of the capturedvideo streams.
 8. The automated process of claim 7 wherein the stillimages are selected from the i-frames of the captured video streams. 9.The automated process of claim 1 wherein at least some of the capturedevices are mobile telephones, and wherein the control device is atablet computer.
 10. The automated process of claim 1 wherein the dataprocessing system is a computer server comprising a processor, memoryand interface to the network, wherein the processor is configured toexecute instructions stored in the memory to perform the receiving,forwarding and encoding.
 11. The automated process of claim 1 whereinthe data processing system is implemented within a cloud processingserver system.
 12. A data processing system to create a video productionstream of an event occurring at a location, the data processing systemcomprising a processor, a memory and an interface to a network, whereinthe memory comprises computer-executable instructions that, whenexecuted by the processor, cause the data processing system to executean automated process comprising: receiving, via a network, a pluralityof captured video streams, wherein each of the captured video stream isa live stream of an event, wherein each captured video stream isreceived by the data processing system via a network from one of aplurality of capture devices operated remotely from the data processingsystem; combining, by the data processing system, the video imagery fromeach of the captured video streams received from the plurality ofcapture devices to create a single combined stream that includes videoimagery from each of the captured video streams; forwarding the singlecombined stream that includes the imagery from each of the capturedvideo streams from the data processing system via the network as aninput stream to a control device located remotely from the dataprocessing system to thereby permit a user of the control device to vieweach of the captured video streams as part of the single combined streamand to select one of the captured video streams from the single combinedstream for output to the video production stream; and in response to acommand received from the control device that identifies the selectedone of the captured video streams from the single combined stream, thedata processing system selecting the selected one of the captured videostreams from the captured video streams received from the plurality ofremotely-operated capture devices and providing only the selected one ofthe captured video streams for output to the video production stream.13. The data processing system of claim 12 wherein the data processingsystem is a cloud-based video processing system operating on hardwarethat is not located at a location of the event, and wherein each of thecaptured video streams are captured by the plurality ofremotely-operated capture devices each located on-site at the locationof the event.
 14. The data processing system of claim 13 wherein theautomated process further comprises compressing the captured videostreams received from the plurality of remotely-operated capture deviceslocated at the location of the event.
 15. The data processing system ofclaim 13 wherein imagery provided in the single combined streamforwarded to the control device comprises a series of still images fromeach of the captured video streams received from the location of theevent.
 16. The data processing system of claim 15 wherein the stillimages are selected from the i-frames of the captured video streams.